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METHOD AND CAMERA FOR IMAGE ENHANCEMENT 
MELD OF THE INVENTION 

The present invention relates to an image enhancement method 
using contextual data. Contextual data means data that provides some information 
5 on the environment or the context in which an image or a series of images were 
captured. While not being limited to such an application, the invention aims in 
particular to enhance images captured with general public type cameras. They are, 
for example^ images captured by means of a digital camera, or by means of any 
other mobile communications equipment integrating a camera function, such as 
10 telephone cameras, camphones, or any other mobile terminal. 

BACKGROUND OF THE INVENTION 
In the above mentioned field, there is a method enabling images to 
be enhanced with data relating to sounds, geographic data or even data reflecting a 
psychological state of the person who took the picture. Such a metihod is described, 
15 for example, in the document (1) whose reference is given at the end of the present 
description. 

Lnplementation of the method entails multimedia equipment, i.e. 
equipment enabling various types of data to be captured and played back on 
various types of media. It can be, for example, hybrid portable equipment 
20 integrating a camera, a display device, a sound pla3^ack device, and an interactive 
interface for a user. Such equipment generally contains a conomon memory for data 
storage. The manory may contain image data captured with the camera, sound data 
from an MP3 type player, and any other data type captured on flie interface by the 
user. 

25 Other data is linked to the image data capable of being played back 

conconndtantly with the display of the images. The additional data relate to sounds, 
music, or, more generally, to the picture-taking drcxmistances. Their playback 
aims to bring to the user's nodnd a broader emotional and perceptual context linked 
to the images. The image data are thus considered as enhanced. 
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The use of hybrid equipment for the simultaneous playback of 
images, music or other contextual elements of the picture-taking, suffers however 
from a number of limitations. 

The most significant limitation is set by the hybrid equipment itself, 
5 and in particular by its intrinsic capacity to play back the pictm-e-taking context. 

For example, if the equipment is only capable of playing back the music, the image 
enhancement remains limited to the musical context. 

A second limitation, linked to the previoxis, relates to the fact that it 
is not possible to implement image enhancement with a camera more summary 
10 than the hybrid equipment. 

SUMMARY OF THE INVENTION 
It is the object of the invention to propose a method of image 
enhancement that does not have the above-mentioned limitations. 

It is an object of the iavention in particular to propose an image 
15 enhancement method capable of being implemented using a simple camera, not 
having other multimedia resources. 

It is also an object of the invention to propose an image 
enhancement method capable of integrating the user's human relational context, i.e. 
to identify the persons present during the picture-taking. 
20 It is further an object of the invention to propose a simplified 

camera enabling the enhancement of captured images. 

To achieve the above-mentioned objects, the invention has more 
specifically for object an automatic method of the enhancement of digital images 
iising contextual data. The method comprises: 
25 - a) searching, in the camera's use environment, for local distribution data 

transmitters, 

- b) the estabUshment of commvmications with the transmitters present in the 
environment, 

- c) the reception of contextual data transmitted by tiie transmitters, and, 
30 - d) the linkmg of at least part of the contextual data to the image data 

c^tured in the use environmCTit. 
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la the meaning of the present inv^tion local distribution data 
transmitter means any device capable of transmitting data by radio or optical 
channels at a distance of some tens of meters. They are, for example, transmitters 
known by the commercial names "Bluetooth" or "Wi-Fi", or even any transmitter 
5 operating according to aradio mode known as IEEE 802.11.x, IEEE 802.15.x, 
UWB, cellular or optical (IRDA), etc. 

The invention method benefits from the fast mxiltiplication of the 
niunber of domestic devices equipped with a local commimications interface. This 
phenomenon results from the user desire to reduce wire cabling between devices 
10 that have to exchange information, and which are generally used in the same room. 
For example, various components of a personal computer, such as the ke}^oard9 
mouse and central processing imit can conmiunicate in this mode. Other devices 
such as mobile phones, tuners, televisions, car radios, multimedia data drives can 
also have a local distribution transmitter. 
15 Local distribution transmitters are capable of transmitting messages, 

whose form can vary according to the communications protocol planned by their 
programming. Nevertheless, fr>r the same device type, the transmission contains an 
amoimt of data that is usual and specific to this device type. 

As an illustration, the local distribution transndtter of a musical 
20 compact disc player generally transmits a message containing information about 

the piece of music that is being played. Additional information such as information 
relating to the music album, the author, the interpreter, etc. can be also be 
distributed. The local distribution transmitter of a mobile phone transnodts 
identification data of the subscriber. This is, for example, his/her name or phone 
25 mmiber. A television can be for example, equipped with a transmitter that 

distributes information about the current program and possibly a URL address to 
the site of the television channel where this program can be found later. 

The gathering of data capable of enhancing images, which 
corresponds to steps a) to c) of the method, can take place at each picture-taking 
30 release. All or part of these steps can also take place advantageously, in a standby 
phase that precedes, and/or follows the release. The standby phase corresponds to 
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an operating state of the camera during which a local communication interface of 
the camera is in operation without necessarily any picture-taking- 

When the surrounding devices capable of supplying contextual data 
are not known, the step of searching for transmitters can comprise the detection of 
5 carrier waves capable of coming jfrom local distribution data transmitters, the 
identification of the communication protocols used by the transmitters and the 
identification of the services available on the transmitters. 

For example, the Bluetooth standard provides for a set of protocol 
components required to identify the available services. For example, the role of 

10 SDP (Service Discovery Protocol) is to discover the available services on the 
surrounding devices. Such services can include file transfer, fax, modem 
connection or data transfer services. In the same way the invention uses a service 
discovery protocol to identify the surrounding devices for which is planned, for 
example, a data transfer service, in this case of contextual data. 

15 After the detection of a communication protocol and/or a service, 

conmnmication can be established between the camera and a transmitter. More 
specifically, the camera can interrogate the transmitters. Interrogation takes place 
either according to a discovery protocol pre-establislied so as to select the 
transmitters respecting this protocol, or according to the protocol previously 

20 determined for each identified transmitter. 

Image ©ohancement is preferably performed according to a mode 
similar to human mnestic and sensory mechanisms. People link sensory 
perceptions in their recollection either because they are concomitant, or because 
there is an emotional, cultural, or rational link between them. In the last case, 

25 concomitance can be imperfect. 

In other words it is convenient to retain data for image 
enhancement, either because it gives ioformation about an event temporarily close 
to the picture-taking moment, or because it gives information about an event 
capable of having a strong link with the .c^tured image for the user. 

30 It is particularly diflScult to.assess such a link. However, it is 

possible to assess, according to tiie type of contextual data transmitter, the 
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emotional force that the context might have, and the mnestic persistence tihat it can 
arouse. For illustration, a user will probably better remember the music that he/she 
heard at the moment he/she took a photograph than the weather data that a local 
meteorological center could provide. Further, the repetition of contextual data 
5 leads to the assumption that the corresponding context had more strongly, 
impressed the user. 

Thus, according to a perfection of the invention, step d) of the 
method can comprise, for any contextual data received, the selection of a valid 
time slot, and the linking of the contextual data to the data of each image captured 
10 in the valid time slot 

Given the previous deliberations, the time slot can advantageoiisly 
be fixed according to the type of device or eqtiipment that the contextual data 
comes from. It can also be fixed according to the repetition firequency of the 
contextual data. 

15 The linking of data to images can take several forms. Very simply 

contextual data can be linked to the image data as metadata. They can then be read 
and decoded at the same time as the image data. 

According to another option, the link can be made using a pointer. 
The pointer is for example a coded graphic array, or barcode, capable of being read 

20 by a digital pen, or a scaimer. In this case, step d) can comprise the recording of 

conte?rtual data in a database and the linking to the contextual data of data coded as 
a graphic array capable of being read by a digital pen or by a scaimer. The data of 
the coded array can be recorded with the image data. Furthermore, a combined 
printing of an image with the image data, and a coded graphic array with the coded 

25 graphic array data can be envisaged. Also the contextual data can be printed 

directiy with the images, when they are capable of being converted into a legible 
format. 

When the array is read and recognized by a digital pen, this issues 
an interrogation command of a remote database, and for data transfer to a suitable 
30 terminal of the user. In this case the command can be used to download sound, 
image or music data. 
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The invention also relates to a camera comprising: 
local communications interface, 

means ofrecognition and interrogation of local distribution data 
transmitters, using the communications interface, and 
5 - mCTiory for recording images linked to data coming jfrom local 

distribution transmitters. 

The camera can also comprise a local distribution data transmitter 
using the local communications interface, so as to transmit identijBcation data to 
other cameras of the same type. 
10 Other charact^stics and advantages of the invention will appear in 

the following description, with reference to the figures in the appended drawings. 
This description is given purely as an illustration and is not limiting. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is an illustration of a data exchange mode between 
15 equipment situated in a camera use environment, for implementing a method of 
image enhancement corresponding to the invention. 

Figure 2 is a flowchart corresponding to the possible 
implementation of an image enhancing method according to th.e invention. 

Figure 3 is a time-flow diagram illustrating a special image 
20 enhandng rule. 

DETAILED DESCRIPTION OF THE INVENTION 
Figure 1 represents a number of domestic equipment items situated 
in an enviroimient involving a user provided with a camera 1 0 according to the 
invention. 

25 Domestic equipment is mainly equipment for the playback or 

capture of soxmd, image or information. In this case it can be a television 12, stereo 
equipment 14, a personal computer 16 and mobile phones 18, 19. Otha: equipment 
could also be present in the enviroimient. 

The above-mentioned domestic equipment, as well as the camera 10 

30 are each equipped with a local conamunications interface. Interfaces are 
symbolically represented as small antennas 20. 
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The \ise eavironment of the camera is defined by the local character 
of the trananission and reception capacities of the coTnrmxni cations interfaces. In 
other words the environment is limited to the transmission range. For known local 
communication interfaces, "Bluetooth" or "WiFi" type, the range is, as previously 
5 mentioned, some tens of meters. It is sufficira.t to cover a room containing the 
user. The use environment modifies over time, because of the user's movements 
with the camCTa. 

It should also be stated that the various equipment items of figure 1 
are not represented according to a uniform scale, for simple reasons of clarity of 
10 the figure. 

Each item of equipment is capable of si^plying an amount of data 
through its communications interface 20. 

For example, the television is capable of transmitting information 
relating to a television program that is being broadcast, or a uniform resource 
15 locator address (URL). This is, for example, the address of an Internet site firom 
which program can be downloaded, or a site where additional information about 
the television program can be obtained, etc. 

Similarly, and already mentioned in the description of the invention, 
the stereo equipment can transmit information relating to a replayed piece of 
20 music, and the mobile phone can transmit an identification of its main user. Thus 
each equipment item is capable of locally distributing information about the visual, 
sound or information context that it generates. This information is again called 
"contextual data". The simplest contextual data that is generally transmitted, is flie 
identification of the nature of the transmittiag equipment. In other words it is 
25 possible to know if the contextual data comes from the television, mobile phone or 
firom some otiier item of equipment 

All the image, audio, video and more generally sensory signals 
captured by all the equipment present and communicating ia the environment 
during the use of the camera, constitute sources of contextual data. 
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The camera 10 comprises, like the other equipment, an antemia for 
local distribution and a local cotimiunications interface. This is formed by a 
module 22 of radio transmission-reception, linked to the antenna 20. 

The local communications interface is run by a control unit 24 
5 programmed to recognize local distribution data transmitters. The unit 24 is 
programmed in particular to detect carrier waves, recognize commLunication 
protocols and establish communication links with the idrntiUBed transmitters. 
"Transmitter" here means any device in the environment capable of supplying 
contextual data. 

10 The communication l^rilrs between the camera 10 and the 

transmittCTS are symbolized by double arrows 30, to indicate a conomunication 
mode according to a question/answer protocol. The communicatioii links can be 
established successively and periodically, or concomitantly. Other commtmication 
protocols, such as protocols for blind data distribution by tranaoiitters can be also 

15 implemented. 

The communication established between the camera and the local 
distribution transmittors enables information to be gathered as contextual data. As 
previously mentioned, this can involve textual data, a uniform resource locator 
address (URL) or, evCTi multimedia data that can be used directty in a suitable 
20 device. 

These data are processed and temporarily saved in a memory 26. 
They can be extracted from it to be linked to image data coming fr-om the camera's 
image sensor 28. While not being represented in great detail on the figure, an 
image processing unit can be linked to the sensor 28 to supply the image data. The 

25 image data and the data extracted from the memory 26 are linked and stored in a 

memory 32 for the temporary storage of enhanced images. On figure 1 the memory 
26 containing image data only and ibe memory 32 containing the enhanced images 
are represented separately. This does not exclude the use of a single memory for 
storing the raw images and enhanced images at the same time. Furthermore, an 

30 additional memory 34 can be platmed for the temporary storage of as yet non- 
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enhanced images. Such a memory in particular enables images to be stored for a 
time necessary to gather contextual data, or finish the gathering. 

The management of contextual data and image data will be 
described in more detail hereafter, with reference to figure 3. 
5 Figure 2, described below, shows the main steps of the image 

enhancement method. Some of these steps were previously mentioned with 
reference to figure 1 . They are thus simply recalled. 

A first operation, shown by reference 100 is searching, in the 
camera's use enviroimient, for local distribution data transmitters. This operation 
10 includes the search for carrier waves coming &om possible transmitters, and the 
identification of the communication mode used by the transmitters found in the 
enviromnent- This amounts to establishing the conmiuiiication protocol used. The 
protocol is idratified firom signatures, i.e. by the recognition of characteristic 
information packets. It can be also determined by trying to establish a 
15 communication according to several prograromed discovery protocols. The camera 
can ia particular detemiine the protocol to be used by trying to establish a 
communication according to, for example, a Bluetooth type protocol or a WiFi 
type protocol. 

The search for local distribution data transmitters takes place in a 
20 camera standby mode, i.e. independentiy of picture-taking. By moving about 
during the day, the user successively moves the camera into or out of the radio 
coverage area of various transmitters. The camera's use environment is thus liable 
to change. 

A second step 102 comprises the establishment of a commuiucation 
25 link and the interrogation of the previously identified trarLsnoitters. The 

coromimication link can be established for a first time with each new transmitter 
identification in the cam^a's use environment. The transmitter can th©a be 
interrogated periodically, for as long as it is in the camera's use environment. The 
camera can also be put in a standby mode in which it simply gathers data 
30 spontaneously transmitted by the transnntters of its environment. Finally, the 
camera can be designed to program the transmitters of its environment by the 
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transmission of a suitable piece of program code. The transmitted program code 
causes, for example, the sending by the environment's transnodtters of contextual 
data for each change of their specific data or at preset time intervals. Specific data 
are data fliat the transmitters transmit or receive. 
5 The reception of data transmitted by the various transmitters is 

shown on figure 2 by the reference 104. Among these data, one can find on the one 
hand, data 1 06 ideatifying the transmitting equipment type, and on the other hand, 
contextual data 108 transmitted by these transmitters. The data identifying the 
transmitter type indicate for example whether the transmitter is a television, 

10 telephone, computer, compact-disc player or some other equipment. These data can 
also be used as contextual data in the rest of the mefliod. 

The transmitter identification data and the contextual data are saved 
and used to check the image CTihancement Reference 110 indicates a saving 
operation and also an available data management operation. It will be described in 

15 more detail with reference to figure 3. 

Returning to figure 2, the block 120 indicates a picture-taking 
release. According to the type of camera used, this can involve a single picture- 
taking release or a picture-taking sequence release. The release causes the capture 
of a set of image data relating to one or more images. These image data can be 

20 formatted and/or temporarily stored. They are, for example, put into a digital file. 

Following these operations the linking 122 of the image data and 
contextual data takes place. The linking can be direct, for example, by registering 
the contextual data as metadata in the files 124 moreover containing the image 
data. 

25 According to another option, the linking of contextual data to the 

image data can be indirect, by simply linking a useful pointer to the image data to 
find the contextual data in a remote database 1 12. The pointer consists for example 
of coded array data 114 capable of being read by means of a digital pen 116. 
Following the reading of the coded array, a reading message corresponding to the 

30 coded array is transmitted fi-om the digital pen to a server or computer. This houses 
a database containing the image's contextual data. The contextual data identified 
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jfrom the read coded array can then be transmitted to the user. When the contextual 
data are themselves addresses or pointers, the otiier data towards which the 
contextual data point can be transmitted or dowiiloaded. This can be multimedia 
data capable of being reproduced at the same time as the images, by means of a 
5 suitable player. Figure 2 illustrates the simultaneous printing on a support 1 1 8 of a 
photograph 1 19 and a coded array 1 14. The support 1 18 is represented with a 
single photograph. It could also contain several photos, in the style of an index 
strip on which each photograph would be linked to its own coded array. The coded 
array data and the image data can be linked and saved in order to be printed out 
10 later. 

Figure 3 illustrates a managemmt option for data coming from local 
distribution transmitters. For reasons of simpUfLcation only three local distribution 
transmitters are considered. They are marked with letters A, B and C. 

The instants of contextual data reception are marked on a tune scale 
15 indicated with the letter t. Instants tj, t2, U and ts correspond to the reception of 

messages Ai, Bi, A2, and Ci. The received messages are symbolized by rectangles 
whose length is proportional to the duration of validity. Here duration of validity 
means the time slot during which contextual data may be held for image 
enhancement. This does not exclude long^ storage of the contextual data, possibly 
20 enabling them to be made available to the user for custom enhancement. 

Letters Di and D2 indicate respectively picture-taking releases at 
instant ta and ts. 

It can be seen on figure 3 that the validity durations are not id^tical 
for the various devices A, B, and C. Data giving information on the transmitter 
25 type, in this case data identified with the reference 108 on figure 2, is used to 
provide the validity duration of the each device's data. 

Another parameter that can be used to provide the validity duration 
is the frequmcy with which an equipment item is used or the frequency with which 
it transmits similar contextual data. 



wo 2005/060239 



PCT/EP2004/013404 



12 

This is illustrated for flie equipment item A, A first contextual data 
Ai is stored witii a first validity duration, A second data A2, more or less identical, 
while coming firom the same equipment item has longer validity duration. 

This enables the fact to be taken into account that the repetition of a 
5 phenomenon or an event has the effect of more strongly marking the user's 

recollection. For example, if the equipment item A is soimd playback equipment 
that transmits a piece of music identified by the contextual data Ai. The 
retransmission of the same piece of music, identified by the data A2, will result in 
longer storage of the data A2 for image enhancCTaent purposes, 

10 The release Di takes place at the instant ta when data Ai and Bi are 

valid. The image captured at the instant ta is thus enhanced with data Ai and Bi. 

The contextual data Ai, A2 and Ci are capable of being used for 
image enhancement, firom the moment they are received, i.e. firom instants ti, t4 and 
t6. Such contextual data managecnent respects a thne causation rule. Respecting a 

15 time causation rule demonstrates the fact that the user more easily Unks in his/her 
recollection concomitant perceptions or perceptions that he/she oreated an 
intellectual link for. For example, the iiser will more easily link an event 
experienced with music that he/she hears or that he/she hears briefly, than with 
music that he/she heard a long time before event occurred. The wedding march 

20 heard at a marriage is a typical illustration. 

For certain equipment however, there is an interest in breaking the 
time causation rule. This is the case for equipment such as the mobile phones 18, 
19 of figure 1. They provide contextual data of the user's social environment 
during picture-taking. The phone nimibers or the subscribers names using them 

25 indeed enable the user to recall which people were aroimd when a photo was taken. 
However, it is quite likely that these people were already together with the user 
before they made use of their telephones or before the telephones were powered 
up. In other words, the reception of contextual data is not necessarily concomitant 
with the perception of the user whose images are required to be enhanced. 

30 It can be seen that the data Bi of figure 3 is active after the instant tz 

of its reception, but also before its reception. This paradoxical breakmg of the time 
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causation rule is possible provided that the image data and the contextual data are 
not linked at the picture-taking moment, but later. An illustration of this is given 
by the release D2. The release D2 at the instant ±5 is used to enhance the image 
captured not only with contextual data valid at the instant ts, but also with 
5 contextual data valid in a time window. The time window, between instants ts and 
ty, is shown by a dot and dash line. The image captured at the instant ts is thus 
enhanced by the contextual data A2 that was valid, and the contextual data Ci, 
valid from instant U only. The instant is indeed included in the time window. It 
can be easily understood that the linking of the image and the contextual data 

10 cannot take place at the picture-taking instant, because at this instant data Ci was 
not available. It can take place with a preset delay, which in this case is greater 
than the time interval ty-te. Everything happens as if the data Ci was abeady valid 
at the instant ts. 
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